PCIe routing

ABSTRACT

A data processing apparatus is provided that includes communication circuitry to transmit an interconnect message to a root port using a physical address mapped to the root port. Translation circuitry encapsulates, within the interconnect message to the root port, a Peripheral Component Interconnect Express (PCIe) message to a destination, the PCIe message having routing information encoded as a PCIe bus number associated with the destination.

TECHNICAL FIELD

The present disclosure relates to data processing.

DESCRIPTION

Messages can be routed in a Peripheral Component Interconnect Express(PCIe) network by providing routing information including the bus numberto which a destination device is linked. This form of routing is oftenrelied on for the transmission of certain types of messages. However, ittraditionally does not work in a system where routing is based onphysical addresses. When a packet has to cross through a system whichhas routing based on physical addresses to travel from a PCIe requestorto another PCIe requestor, the bus number based routing information isnot sufficient.

SUMMARY

Viewed from a first example configuration there is provided a dataprocessing apparatus comprising: communication circuitry configured totransmit an interconnect message to a root port using a physical addressmapped to the root port; and translation circuitry configured toencapsulate, within the interconnect message to the root port, aPeripheral Component Interconnect Express (PCIe) message to adestination, the PCIe message having routing information encoded as aPCIe bus number associated with the destination.

Viewed from a second example configuration there is provided a method ofdata processing comprising: encapsulating, within an interconnectmessage to a root port, a Peripheral Component Interconnect Express(PCIe) message to a destination, the PCIe message having routinginformation encoded as a PCIe bus number associated with thedestination; and transmitting the interconnect message to the root portusing a physical address mapped to the root port

Viewed from a third example configuration there is provided anon-transitory computer-readable medium to store computer-readable codefor fabrication of a data processing apparatus comprising: communicationcircuitry configured to transmit an interconnect message to a root portusing a physical address mapped to the root port; and translationcircuitry configured to encapsulate, within the interconnect message tothe root port, a Peripheral Component Interconnect Express (PCIe)message to a destination, the PCIe message having routing informationencoded as a PCIe bus number associated with the destination.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be described further, by way of example only,with reference to embodiments thereof as illustrated in the accompanyingdrawings, in which:

FIG. 1 shows a pair of compute SoCs connected via an SoC interconnect;

FIG. 2 illustrates a PCIe device in more detail;

FIG. 3 shows an example of root port lookup circuitry;

FIG. 4 shows an example of address lookup circuitry;

FIG. 5 illustrates a flowchart that shows a method of routing for an SoCinterconnect.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Before discussing the embodiments with reference to the accompanyingfigures, the following description of embodiments is provided.

In accordance with one example configuration there is provided a dataprocessing apparatus comprising: communication circuitry configured totransmit an interconnect message to a root port using a physical addressmapped to the root port; and translation circuitry configured toencapsulate, within the interconnect message to the root port, aPeripheral Component Interconnect Express (PCIe) message to adestination, the PCIe message having routing information encoded as aPCIe bus number associated with the destination.

The data processing apparatus interacts with the interconnect usingphysical addresses (e.g. physical addresses of memory mapped to the rootport to which communications can be read and/or written by the rootport). By providing a physical address of a root port, the interconnectmessage can be transmitted to the root port through anintra-chip/inter-chip interconnect. However, the PCIe message that is tobe transmitted to the destination is one that uses a PCIe bus as itsrouting information (i.e. in order to determine the destination of thePCIe message). The translation circuitry therefore maps the destinationPCIe bus to a physical address mapped to a root port, which has thedestination function as part of its PCIe hierarchy and encapsulates thePCIe message within an inter/intra SoC interconnect message to the rootport. From there, the root port is able to transmit the PCIe message toits intended destination. In this way, the PCIe message (which routesaccording to PCIe bus numbers) is able to be transmitted to itsdestination through a network that utilises physical addressing.Consequently, where a peer-to-peer path would take the PCIe messagethrough inter-chip and/or intra-chip interconnection networks to reachthe destination (e.g. through root ports), the message can still berouted correctly. The routing information can be comprised within thePCIe message by being encoded as the PCIe bus number associated with thetarget/destination.

In some examples, the translation circuitry storage comprises: root portlookup circuitry to store correspondences between identifiers of rootports and PCIe buses; and address lookup circuitry to storecorrespondences between root ports and physical addresses. Thetranslation process can therefore be a two-stage process. In a firststage, a root port (e.g. a target root port) is determined for thedestination bus to which the PCIe message is to be transmitted. Then, ina second stage, the physical address associated (e.g. mapped to) withthat root port is determined. This physical address is used as theaddress in the intra-chip/inter-chip interconnect message thatencapsulates the PCIe message.

In some examples, the root port lookup circuitry comprises, for each ofthe root ports, a secondary bus number and a subordinate bus number thatidentifies the highest secondary bus number that is assigned to a linkdownstream of the root port. The bus numbers are assigned at startupduring enumeration process and these numbers may be altered over time inresponse to other devices joining the network. Each device is associatedwith and addressed by the bus number assigned to the PCIe link thatconnects the device to the PCIe fabric. The secondary bus number is thebus number assigned to a root port's or switch's own PCIe link. Asubordinate bus number of a root port or a switch port accounts for allbus numbers assigned to links in its downstream hierarchy. For instance,if a root port is connected to a switch, which has two ports withsecondary bus numbers 11 and 12, then the subordinate bus number of theroot port is 12. The PCIe peripherals connected to the switch ports withsecondary bus numbers 11 and 12 will consume any messages that targetbus numbers 11 and 12 respectively. Note that the IDs in these systemsare assigned in a depth first order.

In some examples, the root port lookup circuitry is configured toidentify the root port whose secondary bus number is less than or equalto the PCIe bus number and whose subordinate bus number is greater thanor equal to the PCIe bus number. Since the bus numbers are assigned in adepth-wise fashion, a root port is responsible for routing messages toany device (switch, peripheral, etc.) that falls within the range of theroot port's own secondary bus number and the subordinate bus number.Therefore, if the bus number of a destination lies between these twonumbers associated with a particular port then that destination is theplace to which the interconnect message containing the PCIe messageshould be transmitted.

In some examples, in response to the interconnect message, the root portlookup circuitry is configured to check the secondary bus number and thesubordinate bus number of each of the root ports against the PCIe busnumber in parallel. By checking in parallel, the identity of thecorresponding root port can be provided more quickly than a system inwhich the root ports are checked sequentially. The check could becarried out in response to receiving the interconnect message or, if theinterconnect message is generated, in response to the message beinggenerated for instance.

In some examples, in response to the PCIe message, the translationcircuitry is configured to look up an identifier of the root port usingthe root port lookup circuitry and then to look up the physical addressmapped to the identifier of the root port using the address lookupcircuitry. The use of the root port ID to identify the physical addresstherefore follows the identification of the destination root port thatis necessary to communicate with the destination of the PCIe message.The looking up of the identifier could occur, for instance, when thePCIe message has been generated or when it is about to be generated.

In some examples, the communication circuitry is configured to write theinterconnect message to the physical address in a memory; and thephysical address in the memory is accessible to the root port. Thememory in question could, for instance, be a memory within the root portitself. However, in other examples, other shared memories could be used.

In some examples, the PCIe message is a PCIe peer-to-peer message. In aPCIe peer-to-peer system, PCIe devices can transfer data with each otherdirectly rather than being requires to exchange messages using a sharedmemory. Peer-to-peer communication can be achieved quickly and with lowlatency because only one write/read occurs rather than two when a sharedmemory is provided. However, peer-to-peer message communication will notwork when the message specifies the destination using the target's busnumber and the message has to cross through an intra-chip/inter-chipinterconnect which has routing based on physical addresses.

In some examples, the routing information comprises a device numberassociated with the destination; and the routing information comprises afunction number associated with the destination. The routing informationcan therefore be a tuple including {the bus number, the device number,the function number}. The bus number indicates the bus on which thedestination device lies. The device number identifies the destinationdevice. Function numbers can be used to ‘virtualise’ physicaldevices—thereby making it possible for a single peripheral to be sharedamong a number of other devices. The tuple therefore makes it possibleto route a message to the correct bus, then the correct physical device,and then the correct virtual device. However, such routing techniquesare not usable in a network were communication takes place usingphysical addresses.

In some examples, the interconnect message is a Vendor Defined Message(VDM). VDMs can be used, for instance, to provide firmware updates toPCIe devices. However, VDMs are often required to be implemented usingpeer-to-peer routing using the PCIe bus number of a target PCIe device.The present technique therefore makes it possible for such messages tobe transmitted by making use of the interconnect message, which proceedswithout the use of peer-to-peer routing.

In some examples, the Peripheral Component Interconnect Express networkis a Compute Express Link (CXL) network. The term ‘PCIe’ is thereforeintended to cover networks, standards, and techniques that are built ontop of PCIe.

Particular embodiments will now be described with reference to thefigures.

FIG. 1 illustrates a system 100 containing a first computesystem-on-chip (SoC) 105 and a second compute system-on-chip (SoC) 110in communication with each other via an SoC interconnect. The systemcontains a number of PCIe devices 125, 130, 150, 160, 170, 185, 195. Thedevices 125, 130, 150, 160, 170, 185, 195 connect to the compute SOCs105, 110 via root ports 115, 135, 155, 165, 175, 190 (which are examplesof the claimed data processing apparatus) and/or PCIe switches 120, 140.Some of the devices 125, 130 are connected to a root port 115 via a PCIeswitch 120. In other cases, a device 150 is directly connected to theroot port 135.

In such a system 100, communication typically occurs from one device 125to another device 170 by using software—either as a consumer or as anintermediary. For instance, the device 125 may write the message into apage (e.g. a 16 kB page) of shared memory 195 and then inform thesoftware. The software would then inform the receiving device 170 thatit should read the data. This software handshaking can be avoided ifdevice 120 sends a message that targets a physical address mapped to thedestination device 170. Such a message will be routed correctly fromroot port 115 to root port 116 through the system interconnect becausethe message uses a physical address to specify the target.

One particular type of message that may be transmitted within such asystem 100 is a vendor defined message (VDM). Such a message can be usedwhere a baseboard management controller (BMC) performs a firmware updateof a PCIe device using VDMs. In this situation, the BMC 125 uses itsinterface to directly send firmware to the device to be updated 170.VDMs expect to be transmitted using peer-to-peer techniques. Withpeer-to-peer communication, a first device 130 can transmit a messagedirectly to a second device 150 without necessitating data being writtento the shared memory 195. This is achieved by providing routinginformation in the form of a destination device identifier and, forinstance, a PCIe bus responsible for the destination device and functionnumber (which allows the physical device 150 to be ‘split’ and thereforeshared). However, such techniques do not work where the packet has totransit through the intra-chip/inter-chip interconnect that uses theabove-mentioned physical addressing. For instance, where thecommunication must travel from one compute SoC 105 to another computeSoC 110, bus number based peer-to-peer communication is typically notpossible and routing instead proceeds via the use of destination'sphysical memory address.

The present technique makes it possible for VDMs to be transmitted in asystem 100 such as that shown in FIG. 1 , where routing is expected tooccur using physical memory addresses.

Also shown in FIG. 1 are bus numbers that are assigned to PCIe linksbetween PCIe devices, switches etc. Each PCIe switch downstream port androot port has three numbers illustrated in FIG. 1 . The first number(the left number) is a “primary bus number”, which is the identifier ofa bus for which the switch or root port is a member (i.e. it is theidentifier of an upstream link). The second number (the middle number)is a “secondary bus number” and is the bus number assigned to thedownstream link of PCIe switch port, or root port in a depth-firstordering. The third number (the right number) is a “subordinate busnumber” and is the highest secondary bus number of any device or switchin that port's downstream hierarchy. The root port or switch downstreamport is responsible for forwarding messages that targets bus numbersthat are between its secondary and subordinate bus numbers to thedownstream hierarchy. Since a PCIe device would be connected to either aroot port or to a switch downstream port via a PCIe link, the bus numberfor a PCIe device would be the secondary bus number of the root port orswitch port to which it is attached. For instance, a root port 175 mayhave a secondary bus number of 7 and a subordinate bus number of 10. Thesubordinate bus number of 10 indicates that the highest secondary busnumber for which the root port 175 is responsible is 10. As illustratedin FIG. 1 , the root port 175 does indeed have responsibility for adevice 185 that is connected to a bus with an ID of 9 and also for adevice 195 that is connected to a bus with an ID of 10.

The secondary bus numbers are established in a process known asenumeration, which can occur when the system 100 is started up for thefirst time. The addition of new devices to the system 100 can cause thebus numbers to change.

Note that although FIG. 1 illustrates an SoC-to-SoC interconnect (aninter-SoC interconnect), the present technique also or alternativelyallows transport over intra-SoC interconnect (which connects differentroot ports within the same SoC die or root ports in different diesinside a package).

FIG. 2 illustrates a device 125 in more detail. In this example, a VDMis received by translation circuitry 200. The VDM could be received fromanother device or the VDM could be generated by the device 125 itself.

In any event, the translation circuitry 200 is responsible forencapsulating the VDM/PCIe message into an interconnect message that issuitable for transmission within the system 100 (i.e. usingaddress-based routing). The translation circuitry 200 contains twocomponents.

The first component is a root port lookup circuit 220, which isresponsible for determining the identity of the root port to which theVDM should be transmitted.

The second component is address lookup circuitry 215, which isresponsible for determining a memory address of a memory in the rootport that has been identified by the root port lookup circuitry 220.

Once the translation circuitry 200 has produced the encapsulatedmessage, communication circuitry 210 is responsible for transmitting theencapsulated message to the memory of the root port. This can beachieved by a memory write operation to the address that has beendetermined using the address lookup circuitry 215. In this way, theencapsulated interconnect message is transmitted to a root port that isresponsible for the destination device specified by the VDM. Once theencapsulated interconnect message is received by the responsible rootport 145, the VDM can be delivered using the routing data includedwithin the VDM (the PCIe bus number, device number, and functionnumber).

FIG. 3 illustrates an example of the root port lookup circuitry 220,which is responsible for determining the root port as that isresponsible for a particular device. A number of entries 305, 310, 315are contained within the lookup circuitry 220. In this example, each ofthe entries can be looked up in parallel. Each entry contains asubordinate bus number and a secondary bus number associated with aparticular root port. In each case, the lookup circuitry 220 determineswhether the target bus number specified in the VDM is less than or equalto the subordinate bus number of an entry and greater than or equal tothe secondary bus number of that entry. This can be achieved by means ofa less than or equal to comparator 320, a greater than or equal tocomparator 325, and a logical AND gate 330. In each case, each entry305, 310, 315 outputs either a ‘0’ if both the conditions are not met ora ‘1’ if either condition is not met.

These outputs can then be combined by an encoder 340 into a one-hotencoding that identifies the root port that is responsible for receivingthe VDM. That is to say that one of the bits of the data output by theencoder 340 is a ‘1’ and other bits are ‘0’. Of course, other encodingsmay also be possible such as a one-cold encoding.

There are a number of ways in which the entries 305, 310, 315 of theroot port lookup circuitry 220 can be populated. However, in someexamples, these entries are populated during the enumeration phase.Repopulation of the entries 305, 310, 315 may occur in response to a newdevice joining the network 100 since this may involve a reallocation ofbus numbers.

FIG. 4 illustrates an example of address lookup circuitry 215. Theaddress lookup circuitry 215 receives, for instance, a one-hot outputproduced by the encoder 340 of the root port lookup circuitry 220. Thedata produced by the encoder 340 of the root port lookup circuitry 220indicates the root port to which the VDM is to be transmitted. Fromhere, it is possible to index into the address lookup circuitry 215 toidentify the address of the page of the memory 195 associated with thatroot port. For instance, if the encoder 340 were to produce an encodingof ‘0010000’, this would indicate that the third root port/address inthe table 215 was to be used. In this case, such identifier wouldindicate the address 0x111FBC00, which would therefore be the page ofmemory to be written to in order to communicate with the third root port(likely root port 145).

Having identified the address to use to communicate with the root port,the VDM message can be written to the relevant page of memory for thatroot port. Once the write is complete, the destination side root portwill be able to read the VDM and route it to the destination deviceusing the routing information contained within the VDM (the PCIe busnumber, device number, and function number)

FIG. 5 illustrates a flowchart 504 that describes a process oftransmitting VDM messages in a system 100 such as that shown in FIG. 1 .As a step 510, the PCIe message (VDM) is obtained. At a step 520, theVDM (its header and its payload) is encapsulated within an interconnectmessage 520. This can be achieved using, for instance, translationcircuitry 200 as previously described. At a step 530, an identity of theroot port that corresponds with a bus number described in the VDM isdetermined. This can be achieved using the root port lookup circuitry220 as previously described. At a step 540, an address associated withthat root port is determined. This can be achieved using the addresslookup circuitry 215 as previously described. Then, at a step 550, thegenerated interconnect message is transmitted by performing a write tothe address that was established in step 514.

In accordance with the above, it is possible for a peer-to-peer messagesuch as a VDM to be routed through an SoC interconnect (which routesusing addresses).

Although the above description has referred to PCIe networks, the abovetechnique is also applicable to other standards that are built on top ofPCIe such as CXL

In the present application, the words “configured to . . . ” are used tomean that an element of an apparatus has a configuration able to carryout the defined operation. In this context, a “configuration” means anarrangement or manner of interconnection of hardware or software. Forexample, the apparatus may have dedicated hardware which provides thedefined operation, or a processor or other processing device may beprogrammed to perform the function. “Configured to” does not imply thatthe apparatus element needs to be changed in any way in order to providethe defined operation.

Although illustrative embodiments of the invention have been describedin detail herein with reference to the accompanying drawings, it is tobe understood that the invention is not limited to those preciseembodiments, and that various changes, additions and modifications canbe effected therein by one skilled in the art without departing from thescope and spirit of the invention as defined by the appended claims. Forexample, various combinations of the features of the dependent claimscould be made with the features of the independent claims withoutdeparting from the scope of the present invention.

We claim:
 1. A data processing apparatus comprising: communicationcircuitry configured to transmit an interconnect message to a physicaladdress mapped to a root port; and translation circuitry configured toencapsulate, within the interconnect message to the root port, aPeripheral Component Interconnect Express (PCIe) message to adestination, the PCIe message having routing information encoded as aPCIe bus number associated with the destination, the translationcircuitry comprising: root port lookup circuitry to storecorrespondences between identifiers of root ports and PCIe buses; andaddress lookup circuitry to store correspondences between root ports andphysical addresses, wherein the root port lookup circuitry comprises,for each of the root ports, a secondary bus number that identifies theroot port and a subordinate bus number that identifies the highestsecondary bus number of any device for which the root port isresponsible; the root port lookup circuitry is configured to identifythe root port whose secondary bus number is less than or equal to thePCIe bus number and whose subordinate bus number is greater than orequal to the PCIe bus number; and in response to the interconnectmessage, the root port lookup circuitry is configured to check thesecondary bus number and the subordinate bus number of each of the rootports against the PCIe bus number in parallel.
 2. The data processingapparatus according to claim 1, wherein in response to the PCIe message,the translation circuitry is configured to look up an identifier of theroot port using the root port lookup circuitry and then to look up thephysical address mapped to the identifier of the root port using theaddress lookup circuitry.
 3. The data processing apparatus according toclaim 1, wherein the communication circuitry is configured to write theinterconnect message to the physical address in a memory; and thephysical address in the memory is accessible to the root port.
 4. Thedata processing apparatus according to claim 3, wherein the memory is amemory of the root port.
 5. The data processing apparatus according toclaim 1, wherein the PCIe message is a PCIe peer-to-peer message.
 6. Thedata processing apparatus according to claim 1, wherein the routinginformation comprises a device number associated with the destination;and the routing information comprises a function number associated withthe destination.
 7. The data processing apparatus according to claim 1,wherein the interconnect message is a Vendor Defined Message (VDM). 8.The data processing apparatus according to claim 1, wherein thePeripheral Component Interconnect Express network is a Compute ExpressLink (CXL) network.
 9. A method of data processing comprising:encapsulating, within an interconnect message to a root port, aPeripheral Component Interconnect Express (PCIe) message to adestination, the PCIe message having routing information encoded as aPCIe bus number associated with the destination; and transmitting theinterconnect message to a physical address mapped to the root port;storing correspondences between identifiers of root ports and PCIebuses, wherein for each of the root ports, a secondary bus number isstored that identifies the root port and a subordinate bus number isstored that identifies the highest secondary bus number of any devicefor which the root port is responsible; storing correspondences betweenroot ports and physical addresses; identifying the root port whosesecondary bus number is less than or equal to the PCIe bus number andwhose subordinate bus number is greater than or equal to the PCIe busnumber; and in response to the interconnect message, the secondary busnumber and the subordinate bus number of each of the root ports ischecked against the PCIe bus number in parallel.
 10. A non-transitorycomputer-readable medium to store computer-readable code for fabricationof a data processing apparatus comprising: communication circuitryconfigured to transmit an interconnect message to a physical addressmapped to the root port; and translation circuitry configured toencapsulate, within the interconnect message to the root port, aPeripheral Component Interconnect Express (PCIe) message to adestination, the PCIe message having routing information encoded as aPCIe bus number associated with the destination, the translationcircuitry comprising: root port lookup circuitry to storecorrespondences between identifiers of root ports and PCIe buses; andaddress lookup circuitry to store correspondences between root ports andphysical addresses, wherein the root port lookup circuitry comprises,for each of the root ports, a secondary bus number that identifies theroot port and a subordinate bus number that identifies the highestsecondary bus number of any device for which the root port isresponsible; the root port lookup circuitry is configured to identifythe root port whose secondary bus number is less than or equal to thePCIe bus number and whose subordinate bus number is greater than orequal to the PCIe bus number; and in response to the interconnectmessage, the root port lookup circuitry is configured to check thesecondary bus number and the subordinate bus number of each of the rootports against the PCIe bus number in parallel.